Search Results: "ivan"

15 October 2014

Raphaël Hertzog: Freexian s second report about Debian Long Term Support

Like last month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In September 2014, 3 contributors have been paid for 11h each. Here are their individual reports: Evolution of the situation Compared to last month, we have gained 5 new sponsors, that s great. We re now at almost 25% of a full-time position. But we re not done yet. We believe that we would need at least twice as many sponsored hours to do a reasonable work with at least the most used packages, and possibly four times as much to be able to cover the full archive. We re now at 39 packages that need an update in Squeeze (+9 compared to last month), and the contributors paid by Freexian did handle 11 during last month (this gives an approximate rate of 3 hours per update, CVE triage included). Open questions Dear readers, what can we do to convince more companies to join the effort? The list of sponsors contains almost exclusively companies from Europe. It s true that Freexian s offer is in Euro but the economy is world-wide and it s common to have international invoices. When Ivan Kohler asked if having an offer in dollar would help convince other companies, we got zero feedback. What are the main obstacles that you face when you try to convince your managers to get the company to contribute? By the way, we prefer that companies take small sponsorship commitments that they can afford over multiple years over granting lots of money now and then not being able to afford it for another year. Thanks to our sponsors Let me thank our main sponsors:

6 June 2014

Gunnar Wolf: What defines an identity?

I must echo John Sullivan's post: GPG keysigning and government identification. John states some very important reasons for people everywhere to verify the identities of those parties they sign GPG keys with in a meaningful way, and that means, not just trusting government-issued IDs. As he says, It's not the Web of Amateur ID Checking. And I'll take the opportunity to expand, based on what some of us saw in Debian, on what this means. I know most people (even most people involved in Free Software development not everybody needs to join a globally-distributed, thousand-people-strong project such as Debian) are not that much into GPG, trust keyrings, or understand the value of a strong set of cross-signatures. I know many people have never been part of a key-signing party. I have been to several. And it was a very interesting experience. Fun, at the beginning at least, but quite tiring at the end. I was part of what could very well constitute the largest KSP ever in DebConf5 (Finland, 2005). Quite awe-inspiring We were over 200 people, all lined up with a printed list on one hand, our passport (or ID card for EU citizens) in the other. Actwally, we stood face to face, in a ribbon-like ring. And, after the basic explanation was given, it was time to check ID documents. And so it began. The rationale of this ring is that every person who signed up for the KSP would verify each of the others' identities. Were anything fishy to happen, somebody would surely raise a voice of alert. Of course, the interaction between every two people had to be quick More like a game than like a real check. "Hi, I'm #142 on the list. I checked, my ID is OK and my fingerprint is OK." "OK, I'm #35, I also printed the document and checked both my ID and my fingerprint are OK." The passport changes hands, the person in front of me takes the unique opportunity to look at a Mexican passport while I look at a Somewhere-y one. And all is fine and dandy. The first interactions do include some chatter while we grab up speed, so maybe a minute is spent Later on, we all get a bit tired, and things speed up a bit. But anyway, we were close to 200 people That means we surely spent over 120 minutes (2 full hours) checking ID documents. Of course, not all of the time under ideal lighting conditions. After two hours, nobody was checking anything anymore. But yes, as a group where we trust each other more than most social groups I have ever met, we did trust on others raising the alarm were anything fishy to happen. And we all finished happy and got home with a bucketload of signatures on. Yay! One year later, DebConf happened in Mexico. My friend Martin Krafft tested the system, perhaps cheerful and playful in his intent but the flaw in key signing parties such as the one I described he unveiled was huge: People join the KSP just because it's a social ritual, without putting any thought or judgement in it. And, by doing so, we ended up dilluting instead of strengthening our web of trust. Martin identified himself using an official-looking ID. According to his recount of the facts, he did start presenting a German ID and later switched to this other document. We could say it was a real ID from a fake country, or that it was a fake ID. It is up to each person to judge. But anyway, Martin brought his Transnational Republic ID document, and many tens of people agreed to sign his key based on it Or rather, based on it plus his outgoing, friendly personality. I did, at least, know perfectly well who he was, after knowing him for three years already. Many among us also did. Until he reached a very dilligent person, Manoj, that got disgusted by this experiment and loudly denounced it. Right, Manoj is known to have strong views, and using fake IDs is (or, at least, was) outside his definition of fair play. Some time after DebConf, a huge thread erupted questioning Martin's actions, as well as questioning what do we trust when we sign an identity document (a GPG key). So... We continued having traditional key signing parties for a couple of years, although more carefully and with more buzz regarding these issues. Until we finally decided to switch the protocol to a better one: One that ensures we do get some more talk and inter-personal recognition. We don't need everybody to cross-sign with everyone else A better trust comes from people chatting with each other and being able to actually pin-point who a person is, what do they do. And yes, at KSPs most people still require ID documents in order to cross-sign. Now... What do I think about this? First of all, if we have not ever talked for at least enough time for me to recognize you, don't be surprised: I won't sign your key or request you to sign mine (and note, I have quite a bad memory when it comes to faces and names). If it's the first conference (or social ocassion) we come together, I will most likely not look for key exchanges either. My personal way of verifying identities is by knowing the other person. So, no, I won't trust a government-issued ID. I know I will be signing some people based on something other than their name, but hey I know many people already who live pseudonymously, and if they choose for whatever reason to forgo their original name, their original name should not mean anything to me either. I know them by their pseudonym, and based on that pseudonym I will sign their identities. But... *sigh*, this post turned out quite long, and I'm not yet getting anywhere ;-) But what this means in the end is: We must stop and think what do we mean when we exchange signatures. We are not validating a person's worth. We are not validating that a government believes who they claim to be. We are validating we trust them to be identified with the (name,mail,affiliation) they are presenting us. And yes, our signature is much more than just a social rite It is a binding document. I don't know if a GPG signature is legally binding anywhere (I'm tempted to believe it is, as most jurisdictions do accept digital signatures, and the procedure is mathematically sound and criptographically strong), but it does have a high value for our project, and for many other projects in the Free Software world. So, wrapping up, I will also invite (just like John did) you to read the E-mail self-defense guide, published by the FSF in honor of today's Reset The Net effort.

5 June 2014

John Sullivan: GPG keysigning and government identification

Please stop recommending that checking government-issued ID is a good way to verify someone's identity before signing their GPG key. Have you been a US bartender before? Or held any other position where you've had to verify an ID? It's not an easy thing to do. People in those positions have books of valid IDs from different states. They have lights that show the security marks. They still get it wrong regularly. A very amateur fake ID, or borrowed real ID, will fool just about everyone in any informal context. What's even worse is that people have a habit of happily looking at passports from other countries than their own, and nodding knowingly. It's fun, but be honest, you have no idea what you're doing. How about just signing keys with people you would actually say you know well enough to trust? It's not the Web of Amateur ID Checking. ID checking is at best ineffective against the threats it's supposed to address, and is probably actually damaging to the Web of Trust because of the false sense of security. No idea what I'm talking about? Learn to encrypt your email by reading the FSF's new Email Self-Defense Guide, published in honor of today's Reset The Net effort.

John Sullivan: GPG keysigning and government identification

Please stop recommending that checking government-issued ID is a good way to verify someone's identity before signing their GPG key. Have you been a US bartender before? Or held any other position where you've had to verify an ID? It's not an easy thing to do. People in those positions have books of valid IDs from different states. They have lights that show the security marks. They still get it wrong regularly. A very amateur fake ID, or borrowed real ID, will fool just about everyone in any informal context. What's even worse is that people have a habit of happily looking at passports from other countries than their own, and nodding knowingly. It's fun, but be honest, you have no idea what you're doing. How about just signing keys with people you would actually say you know well enough to trust? It's not the Web of Amateur ID Checking. ID checking is at best ineffective against the threats it's supposed to address, and is probably actually damaging to the Web of Trust because of the false sense of security. No idea what I'm talking about? Learn to encrypt your email by reading the FSF's new Email Self-Defense Guide, published in honor of today's Reset The Net effort.

11 March 2014

John Sullivan: Drupal UI win

Cancel button next to Cancel link

1 March 2014

John Sullivan: Spritz

Spritz seems like a very interesting way to read quickly. It's the opposite of everything I've read (slowly) about speed reading, which focuses on using peripheral vision and not reading word-by-word. You're supposed to do things like move your eyes straight down the page, taking in whole lines at a time. Interruptions seem like a big problem; interruptions that make me look away, or interruptions in my brain, where I might realize I've not been paying attention for some amount of time. Maybe they should have navigation buttons similar to video players, so you can skip backward 15 seconds at a time. I also do want to go back and review previous pages sometimes for reasons that have nothing to do with interruption, so I wouldn't want word-by-word to be the only way to view a text -- especially when reading nonfiction. I might event want it to work in a mode where you hold down the button on the side of your phone or tablet in order to move the words, and then have them automatically pause when you release. It feels like I'd want a lot of short breaks when reading in this style. It should also be free software, but unfortunately I'm guessing it won't be. I hope someone will make a free software application along these lines -- the basics seem pretty basic.

24 February 2014

Vincent Bernat: Coping with the TCP TIME-WAIT state on busy Linux servers

TL;DR: Do not enable net.ipv4.tcp_tw_recycle. The Linux kernel documentation is not very helpful about what net.ipv4.tcp_tw_recycle does:
Enable fast recycling TIME-WAIT sockets. Default value is 0. It should not be changed without advice/request of technical experts.
Its sibling, net.ipv4.tcp_tw_reuse is a little bit more documented but the language is about the same:
Allow to reuse TIME-WAIT sockets for new connections when it is safe from protocol viewpoint. Default value is 0. It should not be changed without advice/request of technical experts.
The mere result of this lack of documentation is that we find numerous tuning guides advising to set both these settings to 1 to reduce the number of entries in the TIME-WAIT state. However, as stated by tcp(7) manual page, the net.ipv4.tcp_tw_recycle option is quite problematic for public-facing servers as it won t handle connections from two different computers behind the same NAT device, which is a problem hard to detect and waiting to bite you:
Enable fast recycling of TIME-WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).
I will provide here a more detailed explanation in the hope to teach people who are wrong on the Internet. xkcd illustration As a sidenote, despite the use of ipv4 in its name, the net.ipv4.tcp_tw_recycle control also applies to IPv6. Also, keep in mind we are looking at the TCP stack of Linux. This is completely unrelated to Netfilter connection tracking which may be tweaked in other ways1.

About TIME-WAIT state Let s rewind a bit and have a close look at this TIME-WAIT state. What is it? See the TCP state diagram below2: TCP state diagram Only the end closing the connection first will reach the TIME-WAIT state. The other end will follow a path which usually permits to quickly get rid of the connection. You can have a look at the current state of connections with ss -tan:
$ ss -tan   head -5
LISTEN     0  511             *:80              *:*     
SYN-RECV   0  0     192.0.2.145:80    203.0.113.5:35449
SYN-RECV   0  0     192.0.2.145:80   203.0.113.27:53599
ESTAB      0  0     192.0.2.145:80   203.0.113.27:33605
TIME-WAIT  0  0     192.0.2.145:80   203.0.113.47:50685

Purpose There are two purposes for the TIME-WAIT state:
  • The most known one is to prevent delayed segments from one connection being accepted by a later connection relying on the same quadruplet (source address, source port, destination address, destination port). The sequence number also needs to be in a certain range to be accepted. This narrows a bit the problem but it still exists, especially on fast connections with large receive windows. RFC 1337 explains in details what happens when the TIME-WAIT state is deficient3. Here is an example of what could be avoided if the TIME-WAIT state wasn t shortened:
Duplicate segments accepted in another connection
  • The other purpose is to ensure the remote end has closed the connection. When the last ACK is lost, the remote end stays in the LAST-ACK state4. Without the TIME-WAIT state, a connection could be reopened while the remote end still thinks the previous connection is valid. When it receives a SYN segment (and the sequence number matches), it will answer with a RST as it is not expecting such a segment. The new connection will be aborted with an error:
Last ACK lost RFC 793 requires the TIME-WAIT state to last twice the time of the MSL. On Linux, this duration is not tunable and is defined in include/net/tcp.h as one minute:
#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
                                  * state, about 60 seconds     */
There have been propositions to turn this into a tunable value but it has been refused on the ground the TIME-WAIT state is a good thing.

Problems Now, let s see why this state can be annoying on a server handling a lot of connections. There are three aspects of the problem:
  • the slot taken in the connection table preventing new connections of the same kind,
  • the memory occupied by the socket structure in the kernel, and
  • the additional CPU usage.
The result of ss -tan state time-wait wc -l is not a problem per se!

Connection table slot A connection in the TIME-WAIT state is kept for one minute in the connection table. This means, another connection with the same quadruplet (source address, source port, destination address, destination port) cannot exist. For a web server, the destination address and the destination port are likely to be constant. If your web server is behind a L7 load-balancer, the source address will also be constant. On Linux, the client port is by default allocated in a port range of about 30,000 ports (this can be changed by tuning net.ipv4.ip_local_port_range). This means that only 30,000 connections can be established between the web server and the load-balancer every minute, so about 500 connections per second. If the TIME-WAIT sockets are on the client side, such a situation is easy to detect. The call to connect() will return EADDRNOTAVAIL and the application will log some error message about that. On the server side, this is more complex as there is no log and no counter to rely on. In doubt, you should just try to come with something sensible to list the number of used quadruplets:
$ ss -tan 'sport = :80'   awk ' print $(NF)" "$(NF-1) '   \
>     sed 's/:[^ ]*//g'   sort   uniq -c
    696 10.24.2.30 10.33.1.64
   1881 10.24.2.30 10.33.1.65
   5314 10.24.2.30 10.33.1.66
   5293 10.24.2.30 10.33.1.67
   3387 10.24.2.30 10.33.1.68
   2663 10.24.2.30 10.33.1.69
   1129 10.24.2.30 10.33.1.70
  10536 10.24.2.30 10.33.1.73
The solution is more quadruplets5. This can be done in several ways (in the order of difficulty to setup):
  • use more client ports by setting net.ipv4.ip_local_port_range to a wider range,
  • use more server ports by asking the web server to listen to several additional ports (81, 82, 83, ),
  • use more client IP by configuring additional IP on the load balancer and use them in a round-robin fashion,
  • use more server IP by configuring additional IP on the web server6.
Of course, a last solution is to tweak net.ipv4.tcp_tw_reuse and net.ipv4.tcp_tw_recycle. Don t do that yet, we will cover those settings later.

Memory With many connections to handle, leaving a socket open for one additional minute may cost your server some memory. For example, if you want to handle about 10,000 new connections per second, you will have about 600,000 sockets in the TIME-WAIT state. How much memory does it represent? Not that much! First, from the application point of view, a TIME-WAIT socket does not consume any memory: the socket has been closed. In the kernel, a TIME-WAIT socket is present in three structures (for three different purposes):
  1. A hash table of connections, named the TCP established hash table (despite containing connections in other states) is used to locate an existing connection, for example when receiving a new segment. Each bucket of this hash table contains both a list of connections in the TIME-WAIT state and a list of regular active connections. The size of the hash table depends on the system memory and is printed at boot:
    $ dmesg   grep "TCP established hash table"
    [    0.169348] TCP established hash table entries: 65536 (order: 8, 1048576 bytes)
    
    It is possible to override it by specifying the number of entries on the kernel command line with the thash_entries parameter. Each element of the list of connections in the TIME-WAIT state is a struct tcp_timewait_sock, while the type for other states is struct tcp_sock7:
    struct tcp_timewait_sock  
        struct inet_timewait_sock tw_sk;
        u32    tw_rcv_nxt;
        u32    tw_snd_nxt;
        u32    tw_rcv_wnd;
        u32    tw_ts_offset;
        u32    tw_ts_recent;
        long   tw_ts_recent_stamp;
     ;
    struct inet_timewait_sock  
        struct sock_common  __tw_common;
        int                     tw_timeout;
        volatile unsigned char  tw_substate;
        unsigned char           tw_rcv_wscale;
        __be16 tw_sport;
        unsigned int tw_ipv6only     : 1,
                     tw_transparent  : 1,
                     tw_pad          : 6,
                     tw_tos          : 8,
                     tw_ipv6_offset  : 16;
        unsigned long            tw_ttd;
        struct inet_bind_bucket *tw_tb;
        struct hlist_node        tw_death_node;
     ;
    
  2. A set of lists of connections, called the death row , is used to expire the connections in the TIME-WAIT state. They are ordered by how much time left before expiration. It uses the same memory space as for the entries in the hash table of connections. This is the struct hlist_node tw_death_node member of struct inet_timewait_sock.
  3. A hash table of bound ports, holding the locally bound ports and the associated parameters, is used to determine if it is safe to listen to a given port or to find a free port in the case of dynamic bind. The size of this hash table is the same as the size of the hash table of connections:
    $ dmesg   grep "TCP bind hash table"
    [    0.169962] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
    
    Each element is a struct inet_bind_socket. There is one element for each locally bound port. A TIME-WAIT connection to a web server is locally bound to the port 80 and shares the same entry as its sibling TIME-WAIT connections. On the other hand, a connection to a remote service is locally bound to some random port and does not share its entry.
So, we are only concerned by the space occupied by struct tcp_timewait_sock and struct inet_bind_socket. There is one struct tcp_timewait_sock for each connection in the TIME-WAIT state, inbound or outbound. There is one dedicated struct inet_bind_socket for each outbound connection and none for an inbound connection. A struct tcp_timewait_sock is only 168 bytes while a struct inet_bind_socket is 48 bytes:
$ sudo apt-get install linux-image-$(uname -r)-dbg
[...]
$ gdb /usr/lib/debug/boot/vmlinux-$(uname -r)
(gdb) print sizeof(struct tcp_timewait_sock)
 $1 = 168
(gdb) print sizeof(struct tcp_sock)
 $2 = 1776
(gdb) print sizeof(struct inet_bind_bucket)
 $3 = 48
So, if you have about 40,000 inbound connections in the TIME-WAIT state, it should eat less than 10MB of memory. If you have about 40,000 outbound connections in the TIME-WAIT state, you need to account for 2.5MB of additional memory. Let s check that by looking at the output of slabtop. Here is the result on a server with about 50,000 connections in the TIME-WAIT state, 45,000 of which are outbound connections:
$ sudo slabtop -o   grep -E '(^  OBJS tw_sock_TCP tcp_bind_bucket)'
  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
 50955  49725  97%    0.25K   3397       15     13588K tw_sock_TCP            
 44840  36556  81%    0.06K    760       59      3040K tcp_bind_bucket
There is nothing to change here: the memory used by TIME-WAIT connections is really small. If your server need to handle thousands of new connections per second, you need far more memory to be able to efficiently push data to clients. The overhead of TIME-WAIT connections is negligible.

CPU On the CPU side, searching for a free local port can be a bit expensive. The work is done by the inet_csk_get_port() function which uses a lock and iterate on locally bound ports until a free port is found. A large number of entries in this hash table is usually not a problem if you have a lot of outbound connections in the TIME-WAIT state (like ephemeral connections to a memcached server): the connections usually share the same profile, the function will quickly find a free port as it iterates on them sequentially.

Other solutions If you still think you have a problem with TIME-WAIT connections after reading the previous section, there are three additional solutions to solve them:
  • disable socket lingering,
  • net.ipv4.tcp_tw_reuse, and
  • net.ipv4.tcp_tw_recycle.

Socket lingering When close() is called, any remaining data in the kernel buffers will be sent in the background and the socket will eventually transition to the TIME-WAIT state. The application can continue to work immediatly and assume that all data will eventually be safely delivered. However, an application can choose to disable this behaviour, known as socket lingering. There are two flavors:
  1. In the first one, any remaining data will be discarded and instead of closing the connection with the normal four-packet connection termination sequence, the connection will be closed with a RST (and therefore, the peer will detect an error) and will be immediatly destroyed. No TIME-WAIT state in this case.
  2. With the second flavor, if there is any data still remaining in the socket send buffer, the process will sleep when calling close() until either all the data is sent and acknowledged by the peer or the configured linger timer expires. It is possible for a process to not sleep by setting the socket as non-blocking. In this case, the same process happens in the background. It permits the remaining data to be sent during a configured timeout but if the data is succesfully sent, the normal close sequence is run and you get a TIME-WAIT state. And on the other case, you ll get the connection close with a RST and the remaining data is discarded.
In both cases, disabling socket lingering is not a one-size-fits-all solution. It may be used by some applications like HAProxy or Nginx when it is safe to use from the upper protocol point of view. There are good reasons to not disable it unconditionnaly.

net.ipv4.tcp_tw_reuse The TIME-WAIT state prevents delayed segments to be accepted in an unrelated connection. However, on certain conditions, it is possible to assume a new connection s segment cannot be misinterpreted with an old connection s segment. RFC 1323 presents a set of TCP extensions to improve performance over high-bandwidth paths. Among other things, it defines a new TCP option carrying two four-byte timestamp fields. The first one is the current value of the timestamp clock of the TCP sending the option while the second one is the most recent timestamp received from the remote host. By enabling net.ipv4.tcp_tw_reuse, Linux will reuse an existing connection in the TIME-WAIT state for a new outgoing connection if the new timestamp is strictly bigger than the most recent timestamp recorded for the previous connection: an outgoing connection in the TIME-WAIT state can be reused after just one second. How is it safe? The first purpose of the TIME-WAIT state was to avoid duplicate segments to be accepted in an unrelated connection. Thanks to the use of timestamps, such a duplicate segments will come with an outdated timestamp and therefore be discarded. The second purpose was to ensure the remote end is not in the LAST-ACK state because of the lost of the last ACK. The remote end will retransmit the FIN segment until:
  1. it gives up (and tear down the connection), or
  2. it receives the ACK it is waiting (and tear down the connection), or
  3. it receives a RST (and tear down the connection).
If the FIN segments are received in a timely manner, the local end socket will still be in the TIME-WAIT state and the expected ACK segments will be sent. Once a new connection replaces the TIME-WAIT entry, the SYN segment of the new connection is ignored (thanks to the timestamps) and won t be answered by a RST but only by a retransmission of the FIN segment. The FIN segment will then be answered with a RST (because the local connection is in the SYN-SENT state) which will allow the transition out of the LAST-ACK state. The initial SYN segment will eventually be resent (after one second) because there was no answer and the connection will be established without apparent error, except a slight delay: Last ACK lost and timewait reuse It should be noted that when a connection is reused, the TWRecycled counter is increased (despite its name).

net.ipv4.tcp_tw_recycle This mechanism also relies on the timestamp option but affects both incoming and outgoing connections which is handy when the server usually closes the connection first8. The TIME-WAIT state is scheduled to expire sooner: it will be removed after the retransmission timeout (RTO) interval which is computed from the RTT and its variance. You can spot the appropriate values for a living connection with the ss command:
$ ss --info  sport = :2112 dport = :4057
State      Recv-Q Send-Q    Local Address:Port        Peer Address:Port   
ESTAB      0      1831936   10.47.0.113:2112          10.65.1.42:4057    
         cubic wscale:7,7 rto:564 rtt:352.5/4 ato:40 cwnd:386 ssthresh:200 send 4.5Mbps rcv_space:5792
To keep the same guarantees the TIME-WAIT state was providing, while reducing the expiration timer, when a connection enters the TIME-WAIT state, the latest timestamp is remembered in a dedicated structure containing various metrics for previous known destinations. Then, Linux will drop any segment from the remote host whose timestamp is not strictly bigger than the latest recorded timestamp, unless the TIME-WAIT state would have expired:
if (tmp_opt.saw_tstamp &&
    tcp_death_row.sysctl_tw_recycle &&
    (dst = inet_csk_route_req(sk, &fl4, req, want_cookie)) != NULL &&
    fl4.daddr == saddr &&
    (peer = rt_get_peer((struct rtable *)dst, fl4.daddr)) != NULL)  
        inet_peer_refcheck(peer);
        if ((u32)get_seconds() - peer->tcp_ts_stamp < TCP_PAWS_MSL &&
            (s32)(peer->tcp_ts - req->ts_recent) >
                                        TCP_PAWS_WINDOW)  
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
                goto drop_and_release;
         
 
When the remote host is in fact a NAT device, the condition on timestamps will forbid allof the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems. The LAST-ACK state is handled in the exact same way as for net.ipv4.tcp_tw_recycle.

Summary The universal solution is to increase the number of possible quadruplets by using, for example, more server ports. This will allow you to not exhaust the possible connections with TIME-WAIT entries. On the server side, do not enable net.ipv4.tcp_tw_recycle unless you are pretty sure you will never have NAT devices in the mix. Enabling net.ipv4.tcp_tw_reuse is useless for incoming connections. On the client side, enabling net.ipv4.tcp_tw_reuse is another almost-safe solution. Enabling net.ipv4.tcp_tw_recycle in addition to net.ipv4.tcp_tw_reuse is mostly useless. And a final quote by W. Richard Stevens, in Unix Network Programming:
The TIME_WAIT state is our friend and is there to help us (i.e., to let old duplicate segments expire in the network). Instead of trying to avoid the state, we should understand it.

  1. Notably, fiddling with net.netfilter.nf_conntrack_tcp_timeout_time_wait won t change anything on how the TCP stack will handle the TIME-WAIT state.
  2. This diagram is licensed under the LaTeX Project Public License 1.3. The original file is available on this page.
  3. The first work-around proposed in RFC 1337 is to ignore RST segments in the TIME-WAIT state. This behaviour is controlled by net.ipv4.rfc1337 which is not enabled by default on Linux because this is not a complete solution to the problem described in the RFC.
  4. While in the LAST-ACK state, a connection will retransmit the last FIN segment until it gets the expected ACK segment. Therfore, it is unlikely we stay long in this state.
  5. On the client side, older kernels also have to find a free local tuple (source address and source port) for each outgoing connection. Increasing the number of server ports or IP won t help in this case. Linux 3.2 is recent enough to be able to share the same local tuple for different destinations. Thanks to Willy Tarreau for his insight on this aspect.
  6. This last solution may seem a bit dumb since you could just use more ports but some servers are not able to be configured this way. The before last solution can also be quite cumbersome to setup, depending on the load-balancing software, but uses less IP than the last solution.
  7. The use of a dedicated memory structure for sockets in the TIME-WAIT is here since Linux 2.6.14. The struct sock_common structure is a bit more verbose and I won t copy it here.
  8. When the server closes the connection first, it gets the TIME-WAIT state while the client will consider the corresponding quadruplet free and hence may reuse it for a new connection.

7 October 2013

Lucas Nussbaum: talk at Open World Forum

I attended Open World Forum last week (thanks to Inria for funding my travel). It was a fantastic opportunity to meet many people, and to watch great talks. If I had to single out just one talk, it would clearly be John Sullivan s What do you mean you can t Skype?!. On Saturday, I delivered a talk presenting the Debian project. It was my first DPL-ish talk to the general public, so it still needs some tuning, but I think it went quite well (slides available). Next opportunity to talk about Debian: LORIA, Nancy, France, 2013-10-17 13:30 (iPAC seminar).

6 October 2013

Russ Allbery: Review: Captain Vorpatril's Alliance

Review: Captain Vorpatril's Alliance, by Lois McMaster Bujold
Series: Vorkosigan #14
Publisher: Baen
Copyright: November 2012
ISBN: 1-4516-3845-0
Format: Kindle
Pages: 432
This is the fourteenth book in the long-running Vorkosigan series (not counting a few side works). It shifts viewpoint characters, following Ivan Vorpatril instead of either Miles or Cordelia, and could be read on its own since the necessary background is at least briefly explained. But it's probably best read in its proper context in the larger series, since otherwise many of the prior events mentioned here won't have much resonance. Ivan Vorpatril, Miles's cousin, has appeared in multiple previous books along with the rest of Miles's extended family, but he's stayed out of the center of the plot. And he clearly likes it that way. Ivan is not at all ambitious, tries to avoid politics, and just wants to live a quiet life (insofar as that's possible in the Barrayan military), sleep with a few women, and stay out of trouble. But given his relations and his job as aide-de-camp to an admiral this doesn't always work. In this book, the not working comes in the form of Byerly, ImpSec (Imperial Security, the Barrayan equivalent of the CIA), and a request to find out more about a woman who seems to be the target of a bounty. Fairly quickly, Ivan finds himself hopelessly entangled in both politics and in someone else's complex life, and is not entirely sure what to do about either. It's been months since I read this book and I'm only now writing a review, in part because I find it hard to know what to say about it. While Ivan doesn't have Miles's manic energy, the tone of this book is very similar to the rest of the Vorkosigan series, as is the plot and most of the characters. Ivan tries a more laid-back approach and doesn't actively seek danger the way that Miles does, but his persistent efforts to do the right thing in the face of unexpected events lead mostly to the same places. There's a bit of humor, a bit of danger, some galactic politics, some scheming, and a slow process of understanding other people better in short, Captain Vorpatril's Alliance is very similar to the thirteen books that proceeded it. This is obviously not a bad thing. If you've read this far in the series, more of the same is probably just fine. I enjoyed the book. Bujold is great at telling stories, the plot mostly moved right along, and I quite liked the two women Ivan meets over the course of the story. But there wasn't much about the story that grabbed me or, I suspect, that I will find particularly memorable. One reason for my somewhat "meh" reaction is that the core of this story is a romance, and while much of that romance is well-done, there is a large segment in the middle of the book where the suspense of the plot derives mostly from one of the two characters refusing to confide in the other. This is Bujold, so this isn't as much of a train wreck as it might be in the hands of another author, and it is for valid plot reasons. But even with those justifications, I never like that plot element. There were about a hundred pages where I wanted to grab Tej and shake some sense into her, which doesn't endear a character to me. And the leads in a romance, at least this sort of romance, should be endearing. I think one reason why Bujold went for that approach to building tension is that there weren't a lot of options available. The basic outlines of what's going to happen are obvious from early in the book and essentially forced by the structure of the series. We know from past books that Ivan is both trustworthy and related to a staggering variety of powerful people, almost all of whom are highly competent and honorable. It's fairly obvious that, as soon as the necessary people trust Ivan and let him and his family solve their problems, all the problems will be solved. It's also fairly obvious how the romances will play out, and that all the new characters will end up a little in awe of how competent and trustworthy all the Barrayans are (a few misdirections aside). Therefore, much of the book, for a long-time reader of the series, involves waiting until those things happen, and meanwhile getting increasingly annoyed at characters who refuse to trust obviously trustworthy people. All that said, I still quite liked the beginning of the book, because Tej and Rish are a wonderful pair and have a great interpersonal dynamic, and the very end of the book once everyone gets past the communication problems and the plot can deliver on its promises. Both together make up for some noticeable sagging in the middle section of the book. Captain Vorpatril's Alliance is not up to the standards of the most memorable books of the series, but it's a solid dose of "more of the same" and provides its money's worth of entertainment. I'm still hoping for more Ekaterin and more fireworks in the next book, though. Rating: 7 out of 10

1 September 2013

Gregor Herrmann: RC bugs 2013/35

& another week of perl (5.18) related bugs has gone by:

Russ Allbery: Review: Redshirts

Review: Redshirts, by John Scalzi
Publisher: Tor
Copyright: June 2012
ISBN: 1-4299-6360-3
Format: Kindle
Pages: 317
The next few reviews deserve a small apology in advance. Normally, I try to write reviews within a few weeks, or at most a month, after reading a book, while it's still fresh in my mind. This is particularly important since I read a lot and sometimes don't have a good memory for specific detail. But this summer has made it completely impossible to keep to any sort of regular writing schedule. As a result, this book, and the subjects of the next few reviews that follow, are books that I read four months ago and am reviewing from memory. Hopefully, I'll still get the details right. Ensign Andrew Dahl is one of the newest crew members of the Universal Union Capital Ship Intrepid. It's his first posting, just out of the Academy. He's one of five new ensigns, expecting a normal tour of duty (albeit with the pride of being on a capital ship of the fleet). But very shortly after arriving on the ship, Andrew and and the other new ensigns discover that something is very... strange. The casualty rate on the ship is surprisingly high (as foreshadowed in a prologue). The other crew members avoid the bridge crew, and particularly away missions, with startling effectiveness (and desperation). And things regularly happen on the ship that, for lack of a better description, simply make no sense. Redshirts starts off as a very obvious Star Trek parody. So obvious, in fact, that it's eyeroll-inducing. The prologue reads exactly like a section of a Star Trek episode written by a third-rate hack, leavened with some of the humor that readers of Scalzi's blog have come to expect. But it's obvious that Scalzi is doing this intentionally, and it doesn't take long before the subject of the book shifts. It's a Star Trek parody that functions much more like a Star Trek parody than any universe should, to the degree that it becomes obvious to the characters. And the characters, despite being straight from central casting, are clearly not expecting their universe to act that way and find this extremely strange. By far the best part of this book is the second quarter: the characters have been introduced, they're starting to figure out what's going on, and they have to do so while dodging away-team duty (or trying to survive it) and participating in the ridiculous plot contrivances that keep occurring. Scalzi mixes a feeling of serious consequences with wry humor and handles the mix deftly. I was both laughing and eagerly reading to see what comes next. The last half of the book, though, has problems. How many problems the reader perceives will, I think, depend on how hard one thinks about the situation Scalzi has set up, or how much one has thought about similar constructs in the past. But I'm afraid the setup does not survive serious scrutiny in ways that, for me at least, seriously damaged my ability to engage with the story. I'm going to avoid saying explicitly where Scalzi goes with this plot, since it is a spoiler (although it's commonly mentioned in discussions of this book on the Internet). But it happens to be an idea that some friends of mine and I pounded on for years in some of our role-playing scenarios, poking at the corners and implications, and subsequently gave up on because it doesn't work for storytelling. The idea is compelling, and on first glance seems like a great story seed. But it's too large. The problem with really big ideas with huge implications is that they can change the nature of a story so much that there's no longer any point of commonality for the reader and the entire nature of story falls apart. And I think that's what happens here. Scalzi handles the deeper implications of the concept by, essentially, ignoring them. He focuses on one narrow set of implications for one specific set of people, largely skips over any discussion of anything larger, and tries a pivot from wry humor (and a bit of slapstick) into serious emotional depth. For me, in part because I'm so familiar with the concept, none of this really worked. The implications of the discovery that drives this story are so much broader and so much more significant than the characters seem to admit that it threw me right out of the story. The ethical implications are absolutely paralyzing, to the point that I don't think you can write a coherent story about them, but ignoring them doesn't work either. It's the kind of discovery that requires action of some kind beyond the attempt by the characters to go back to what they consider a normal life, even if I have a hard time imagining what that action could be. To be fair, he does try to salvage this in the first epilogue, which is all about the implications of his idea and its effects. But it's still too small. That epilogue just barely touches on the implications on a single person, but if one thinks a bit longer, one starts coming up with more and more implications for all of society, for how we talk about story, for how we raise children, for nearly every form of creativity. Implications that involve nasty concepts like slavery and mind control and murder and the nature of consent in ways that are very difficult to avoid. I think Scalzi's attempts at serious emotional connection, including in the other two attached epilogues (which are perspective-shifting close focuses on some of the characters), also suffer from his world setup, but in a different way. The characters are simply too bland to sustain the level of emotional connection he's striving for. They're bland for a very specific reason that is entirely justified by the story that Scalzi is handling, and the implications of that for the primary protagonist were, I thought, one of the best-handled moments of the book. But they're still bland, which undermines the attempted pivot. All that being said, this is not a bad book. Parts of it, particularly the second quarter, are excellent. And even when I wasn't engaging with the characters, Scalzi's writing on a paragraph level is snappy and often quite funny. The first epilogue in particular is hilarious, even if I think it was too limited in the scope of implications that it considers and a bit too meta in where it goes with those implications. And the story is a fun romp, with drama and humor and tension and a fairly satisfying ending. Everyone's a bit too bland and earnest when Scalzi isn't being funny, but he's funny enough that I didn't mind that much. Despite the relatively low rating, I actually recommend this book. If you don't dive into the metaphysical implications, it's a lot of fun, and if you've thought a lot about this particular set of metaphysical implications, it's somewhat satisfying to argue with (and to internally point out all of the hard questions that Scalzi is skipping over). It's particularly fun if you've watched a lot of Star Trek, since Scalzi gets some of the absurdities note-perfect. But be aware that the character reactions to Scalzi's concept don't hold up to much scrutiny. Rating: 7 out of 10

5 July 2013

John Sullivan: I think this says something about my taste in music

johnsu01@myles:~$ mpc playlist grep -i outro wc -l
14

2 July 2013

Ond&#345;ej &#268;ert&iacute;k: My impressions from the SciPy 2013 conference

I have attended the SciPy 2013 conference in Austin, Texas. Here are my impressions.

Number one is the fact that the IPython notebook was used by pretty much everyone. I use it a lot myself, but I didn't realize how ubiquitous it has become. It is quickly becoming the standard now. The IPython notebook is using Markdown and in fact it is better than Rest. The way to remember the "[]()" syntax for links is that in regular text you put links into () parentheses, so you do the same in Markdown, and append [] for the text of the link. The other way to remember is that [] feel more serious and thus are used for the text of the link. I stressed several times to +Fernando Perez and +Brian Granger how awesome it would be to have interactive widgets in the notebook. Fortunately that was pretty much preaching to the choir, as that's one of the first things they plan to implement good foundations for and I just can't wait to use that.

It is now clear, that the IPython notebook is the way to store computations that I want to share with other people, or to use it as a "lab notebook" for myself, so that I can remember what exactly I did to obtain the results (for example how exactly I obtained some figures from raw data). In other words --- instead of having sets of scripts and manual bash commands that have to be executed in particular order to do what I want, just use IPython notebook and put everything in there.

Number two is that how big the conference has become since the last time I attended (couple years ago), yet it still has the friendly feeling. Unfortunately, I had to miss a lot of talks, due to scheduling conflicts (there were three parallel sessions), so I look forward to seeing them on video.

+Aaron Meurer and I have done the SymPy tutorial (see the link for videos and other tutorial materials). It's been nice to finally meet +Matthew Rocklin (very active SymPy contributor) in person. He also had an interesting presentation
about symbolic matrices + Lapack code generation. +Jason Moore presented PyDy.
It's been a great pleasure for us to invite +David Li (still a high school student) to attend the conference and give a presentation about his work on sympygamma.com and live.sympy.org.

It was nice to meet the Julia guys, +Jeff Bezanson and +Stefan Karpinski. I contributed the Fortran benchmarks on the Julia's website some time ago, but I had the feeling that a lot of them are quite artificial and not very meaningful. I think Jeff and Stefan confirmed my feeling. Julia seems to have quite interesting type system and multiple dispatch, that SymPy should learn from.

I met the VTK guys +Matthew McCormick and +Pat Marion. One of the keynotes was given by +Will Schroeder from Kitware about publishing. I remember him stressing to manage dependencies well as well as to use BSD like license (as opposed to viral licenses like GPL or LGPL). That opensource has pretty much won (i.e. it is now clear that that is the way to go).

I had great discussions with +Francesc Alted, +Andy Terrel, +Brett Murphy, +Jonathan Rocher, +Eric Jones, +Travis Oliphant, +Mark Wiebe, +Ilan Schnell, +St fan van der Walt, +David Cournapeau, +Anthony Scopatz, +Paul Ivanov, +Michael Droettboom, +Wes McKinney, +Jake Vanderplas, +Kurt Smith, +Aron Ahmadia, +Kyle Mandli, +Benjamin Root and others.


It's also been nice to have a chat with +Jason Vertrees and other guys from Schr dinger.

One other thing that I realized last week at the conference is that pretty much everyone agreed on the fact that NumPy should act as the default way to represent memory (no matter if the array was created in Fortran or other code) and allow manipulations on it. Faster libraries like Blaze or ODIN should then hook themselves up into NumPy using multiple dispatch. Also SymPy would then hook itself up so that it can be used with array operations natively. Currently SymPy does work with NumPy (see our tests for some examples what works), but the solution is a bit fragile (it is not possible to override NumPy behavior, but because NumPy supports general objects, we simply give it SymPy objects and things mostly work).

Similar to this, I would like to create multiple dispatch in SymPy core itself, so that other (faster) libraries for symbolic manipulation can hook themselves up, so that their own (faster) multiplication, expansion or series expansion would get called instead of the SymPy default one implemented in pure Python.

Other blog posts from the conference:

17 June 2013

John Sullivan: M-x spook

In light of the recent leaks about the NSA's illegal spying, I've decided to go back to using M-x spook output in my email signatures. cypherpunk anthrax John Kerry rail gun security plutonium Guantanamo wire transfer JPL number key military MD5 SRI FIPS140 Uzbekistan

16 April 2013

Stefano Zacchiroli: bits from the DPL for March-April 2013

Dear Project Members,
   "Now that I have your attention, I would like to make the following
delegations:"

... nah, scrap that. In my last day in office I first of all owe you a report of DPL activities for the last reporting period of this term, i.e. March 8th until today. Here it is! Highlights Talks Over the past month or so I've attended and spoken on behalf of Debian in the following occasions: Assets I've approved the budget for the following forthcoming sprints: Also, we've bought a 3-year warranty pack for the disk array that powers ftp-master.d.o (~900 USD). On the income side, Brian Gupta has started an interesting matching fund experiment, in order to raise funds for the forthcoming DebConf13. The matching fund will be open until April 30th, so your help in spreading news would be welcome. Many thanks to Brian for the idea and to his company, Brandorr Group, for funding it. DPL helpers Three more DPL helpers IRC meetings have been held; minutes are available at the usual place. Legal Spring Cleaning I've finally cleaned up the pile of pending legal matters (but I'm sure new ones will show up for the delight of the next DPL :-P) Once again, I'd like to thank SFLC for the pro bono and very high quality legal advice they keep on offering to Debian. Miscellaneous
Now, before I get sentimental, let me thank Gergely, Lucas, and Moray for running in the recently concluded DPL election. Only thinking of running and then go through a campaign denote a very high commitment to the Project; we should all be thankful to them. Then I'd like to congratulate Lucas for his election. I've known him for a long time, and I can testify about his clear vision of the role Debian has to play in Free Software and on what Debian needs to improve to do so. Best wishes for the term ahead, Lucas! Finally, I'd like to thank you all for the support you've shown me over the past 3 years. Serving as DPL is a great honor, but also a very demanding job. Thank to you all, and to how cool Debian is, it has been for me an incredibly rewarding experience. I had no idea what I were doing when I embarked on this adventure, but in hindsight I don't regret any of it. See you around, as I don't plan to be anywhere far away from Debian anytime soon. Cheers.
PS the day-to-day activity logs for March and April 2013 are available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.20130 3,4

30 March 2013

John Sullivan: Vegan in Amsterdam?

I'll be traveling to Amsterdam next week for a free software conference. Does anyone have recommendations for restaurants that are vegan-friendly? Natural food stores? I'll be staying very near the Central Station.

16 March 2013

Stefano Zacchiroli: bits from the DPL for February 2013 and a half

Dear project members, here's another report of DPL activities, this time for a period longer than usual (February + 1st week of March), so that the next one will be at the very end of the current DPL term. Highlights Appointments DPL helpers Two more DPL helpers IRC meetings have happened, minutes and logs of both are available. Assets Events Past At the beginning of February, I've attended FOSDEM 2013, together with many other Debian people. I didn't have any specific talk this year, but it's been a chance to talk F2F about several ongoing issues (see logs), and help mediating in some conflicts. I've also accepted the invitation to participate in the GNOME Advisory Board meeting, together with Laurent Bigonville of our GNOME team. No report of that has been prepared as of yet (sorry about that), but we have both reported "live" to the rest of the team on IRC. Future Miscellaneous A couple of months ago I've mentioned that I had filed an application, as Debian representative, to participate in a working table to define software procurement rules for the Italian public administration. Good news: my application has been accepted, together with those of other well-known FOSS communities and organizations (e.g. KDE, FSFE). I'll keep you posted of how it goes. Let's go back to elect a new DPL and release Wheezy now,
Cheers.
PS the day-to-day activity logs for February and March 2013 are available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.20130 2,3

18 February 2013

John Sullivan: SCALE

I will be speaking at the Southern California Linux Expo (and yes, given the topics covered, it's missing a GNU). My talk, "Four Freedoms for Freedom," is on Sunday, February 24, 2013 from 16:30 to 17:30.
The most obvious people affected by all four of the freedoms that define free software are the programmers. They are the ones who will likely want to -- and are able to -- modify software running on their computers. But free software is a movement to advance and defend freedom for anyone and everyone using any computing device, not just programmers. In many countries now, given the ubiquity of tablets, phones, laptops and desktops, "anyone and everyone using any computing device" means nearly all citizens. But new technological innovations in these areas keep coming with new restrictions, frustrating and controlling users even while creating a perception of empowerment. The Free Software Foundation wants to gain the support and protect the interests of everyone, not just programmers. How do we reach people who have no intention of ever modifying a program, and how do we help them?
Other presentations on my list to check out (in chronological order, some conflicting): If you will be there and want to meet up, drop me a line.

28 January 2013

John Sullivan: FOSDEM

I'll be at FOSDEM again this year, arriving in Brussels on Thursday 31st and leaving on Tuesday 5th. I'll be speaking on Sunday in the legal issues devroom at 10:00. If you will be there and want to meet up, let me know. I may be trying to watch the Super Bowl from there, a plan that didn't quite work out last year but seems more likely this year. State of the GNUnion FSF licensing policy challenges in 2013 This talk will cover the main challenges facing the Free Software Foundation's Licensing and Compliance lab in 2013, and will invite discussion of the FSF's work and policies in this area. We'll explore:

12 January 2013

John Sullivan: Aaron Swartz

Aaron was an inspiration to me personally, politically, and professionally ever since we met (ice cream and word games with a small group in a bank vault at Herrell's in Harvard Square) several years ago. I don't understand how things got to this point, but I know I'm angry along with Lessig. I'm so sorry for all of his family and friends; all the rest of us can do is try to make even a tiny sliver of the difference he did.

Next.

Previous.